Anatomy segmentation in laparoscopic surgery: comparison of machine learning and human expertise - an experimental study.

BACKGROUND: Lack of anatomy recognition represents a clinically relevant risk in abdominal surgery. Machine learning (ML) methods can help identify visible patterns and risk structures; however, their practical value remains largely unclear. MATERIALS AND METHODS: Based on a novel dataset of 13 195 laparoscopic images with pixel-wise segmentations of 11 anatomical structures, we developed specialized segmentation models for each structure and combined models for all anatomical structures using two state-of-the-art model architectures (DeepLabv3 and SegFormer) and compared segmentation performance of algorithms to a cohort of 28 physicians, medical students, and medical laypersons using the example of pancreas segmentation. RESULTS: Mean Intersection-over-Union for semantic segmentation of intra-abdominal structures ranged from 0.28 to 0.83 and from 0.23 to 0.77 for the DeepLabv3-based structure-specific and combined models, and from 0.31 to 0.85 and from 0.26 to 0.67 for the SegFormer-based structure-specific and combined models, respectively. Both the structure-specific and the combined DeepLabv3-based models are capable of near-real-time operation, while the SegFormer-based models are not. All four models outperformed at least 26 out of 28 human participants in pancreas segmentation. CONCLUSIONS: These results demonstrate that ML methods have the potential to provide relevant assistance in anatomy recognition in minimally invasive surgery in near-real-time. Future research should investigate the educational value and subsequent clinical impact of the respective assistance systems.

Artificial Intelligence for context-aware surgical guidance in complex robot-assisted oncological procedures: An exploratory feasibility study.

INTRODUCTION: Complex oncological procedures pose various surgical challenges including dissection in distinct tissue planes and preservation of vulnerable anatomical structures throughout different surgical phases. In rectal surgery, violation of dissection planes increases the risk of local recurrence and autonomous nerve damage resulting in incontinence and sexual dysfunction. This work explores the feasibility of phase recognition and target structure segmentation in robot-assisted rectal resection (RARR) using machine learning. MATERIALS AND METHODS: A total of 57 RARR were recorded and subsets of these were annotated with respect to surgical phases and exact locations of target structures (anatomical structures, tissue types, static structures, and dissection areas). For surgical phase recognition, three machine learning models were trained: LSTM, MSTCN, and Trans-SVNet. Based on pixel-wise annotations of target structures in 9037 images, individual segmentation models based on DeepLabv3 were trained. Model performance was evaluated using F1 score, Intersection-over-Union (IoU), accuracy, precision, recall, and specificity. RESULTS: The best results for phase recognition were achieved with the MSTCN model (F1 score: 0.82 ± 0.01, accuracy: 0.84 ± 0.03). Mean IoUs for target structure segmentation ranged from 0.14 ± 0.22 to 0.80 ± 0.14 for organs and tissue types and from 0.11 ± 0.11 to 0.44 ± 0.30 for dissection areas. Image quality, distorting factors (i.e. blood, smoke), and technical challenges (i.e. lack of depth perception) considerably impacted segmentation performance. CONCLUSION: Machine learning-based phase recognition and segmentation of selected target structures are feasible in RARR. In the future, such functionalities could be integrated into a context-aware surgical guidance system for rectal surgery.

Prediction of clinically relevant postoperative pancreatic fistula using radiomic features and preoperative data.

Clinically relevant postoperative pancreatic fistula (CR-POPF) can significantly affect the treatment course and outcome in pancreatic cancer patients. Preoperative prediction of CR-POPF can aid the surgical decision-making process and lead to better perioperative management of patients. In this retrospective study of 108 pancreatic head resection patients, we present risk models for the prediction of CR-POPF that use combinations of preoperative computed tomography (CT)-based radiomic features, mesh-based volumes of annotated intra- and peripancreatic structures and preoperative clinical data. The risk signatures were evaluated and analysed in detail by visualising feature expression maps and by comparing significant features to the established CR-POPF risk measures. Out of the risk models that were developed in this study, the combined radiomic and clinical signature performed best with an average area under receiver operating characteristic curve (AUC) of 0.86 and a balanced accuracy score of 0.76 on validation data. The following pre-operative features showed significant correlation with outcome in this signature ([Formula: see text]) - texture and morphology of the healthy pancreatic segment, intensity volume histogram-based feature of the pancreatic duct segment, morphology of the combined segment, and BMI. The predictions of this pre-operative signature showed strong correlation (Spearman correlation co-efficient, [Formula: see text]) with the intraoperative updated alternative fistula risk score (ua-FRS), which is the clinical gold standard for intraoperative CR-POPF risk stratification. These results indicate that the proposed combined radiomic and clinical signature developed solely based on preoperatively available clinical and routine imaging data can perform on par with the current state-of-the-art intraoperative models for CR-POPF risk stratification.

The image-based preoperative fistula risk score (preFRS) predicts postoperative pancreatic fistula in patients undergoing pancreatic head resection.

Clinically relevant postoperative pancreatic fistula (CR-POPF) is a common severe surgical complication after pancreatic surgery. Current risk stratification systems mostly rely on intraoperatively assessed factors like manually determined gland texture or blood loss. We developed a preoperatively available image-based risk score predicting CR-POPF as a complication of pancreatic head resection. Frequency of CR-POPF and occurrence of salvage completion pancreatectomy during the hospital stay were associated with an intraoperative surgical (sFRS) and image-based preoperative CT-based (rFRS) fistula risk score, both considering pancreatic gland texture, pancreatic duct diameter and pathology, in 195 patients undergoing pancreatic head resection. Based on its association with fistula-related outcome, radiologically estimated pancreatic remnant volume was included in a preoperative (preFRS) score for POPF risk stratification. Intraoperatively assessed pancreatic duct diameter (p < 0.001), gland texture (p < 0.001) and high-risk pathology (p < 0.001) as well as radiographically determined pancreatic duct diameter (p < 0.001), gland texture (p < 0.001), high-risk pathology (p = 0.001), and estimated pancreatic remnant volume (p < 0.001) correlated with the risk of CR-POPF development. PreFRS predicted the risk of CR-POPF development (AUC = 0.83) and correlated with the risk of rescue completion pancreatectomy. In summary, preFRS facilitates preoperative POPF risk stratification in patients undergoing pancreatic head resection, enabling individualized therapeutic approaches and optimized perioperative management.

Radiomics-based tumor phenotype determination based on medical imaging and tumor microenvironment in a preclinical setting.

BACKGROUND AND PURPOSE: Radiomics analyses have been shown to predict clinical outcomes of radiotherapy based on medical imaging-derived biomarkers. However, the biological meaning attached to such image features often remains unclear, thus hindering the clinical translation of radiomics analysis. In this manuscript, we describe a preclinical radiomics trial, which attempts to establish correlations between the expression of histological tumor microenvironment (TME)- and magnetic resonance imaging (MRI)-derived image features. MATERIALS & METHODS: A total of 114 mice were transplanted with the radioresistant and radiosensitive head and neck squamous cell carcinoma cell lines SAS and UT-SCC-14, respectively. The models were irradiated with five fractions of protons or photons using different doses. Post-treatment T1-weighted MRI and histopathological evaluation of the TME was conducted to extract quantitative features pertaining to tissue hypoxia and vascularization. We performed radiomics analysis with leave-one-out cross validation to identify the features most strongly associated with the tumor's phenotype. Performance was assessed using the area under the curve (AUCValid) and F1-score. Furthermore, we analyzed correlations between TME- and MRI features using the Spearman correlation coefficient ρ. RESULTS: TME and MRI-derived features showed good performance (AUCValid,TME = 0.72, AUCValid,MRI = 0.85, AUCValid,Combined=0.85) individual tumor phenotype prediction. We found correlation coefficients of ρ=-0.46 between hypoxia-related TME features and texture-related MRI features. Tumor volume was a strong confounder for MRI feature expression. CONCLUSION: We demonstrated a preclinical radiomics implementation and notable correlations between MRI- and TME hypoxia-related features. Developing additional TME features may help to further unravel the underlying biology.

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

Comprehensive Analysis of Tumour Sub-Volumes for Radiomic Risk Modelling in Locally Advanced HNSCC.

Imaging features for radiomic analyses are commonly calculated from the entire gross tumour volume (GTV entire). However, tumours are biologically complex and the consideration of different tumour regions in radiomic models may lead to an improved outcome prediction. Therefore, we investigated the prognostic value of radiomic analyses based on different tumour sub-volumes using computed tomography imaging of patients with locally advanced head and neck squamous cell carcinoma. The GTV entire was cropped by different margins to define the rim and the corresponding core sub-volumes of the tumour. Subsequently, the best performing tumour rim sub-volume was extended into surrounding tissue with different margins. Radiomic risk models were developed and validated using a retrospective cohort consisting of 291 patients in one of the six Partner Sites of the German Cancer Consortium Radiation Oncology Group treated between 2005 and 2013. The validation concordance index (C-index) averaged over all applied learning algorithms and feature selection methods using the GTVentire achieved a moderate prognostic performance for loco-regional tumour control (C-index: 0.61 ± 0.04 (mean ± std)). The models based on the 5 mm tumour rim and on the 3 mm extended rim sub-volume showed higher median performances (C-index: 0.65 ± 0.02 and 0.64 ± 0.05, respectively), while models based on the corresponding tumour core volumes performed less (C-index: 0.59 ± 0.01). The difference in C-index between the 5 mm tumour rim and the corresponding core volume showed a statistical trend (p = 0.10). After additional prospective validation, the consideration of tumour sub-volumes may be a promising way to improve prognostic radiomic risk models.

Comparison of patient stratification by computed tomography radiomics and hypoxia positron emission tomography in head-and-neck cancer radiotherapy.

BACKGROUND AND PURPOSE: Hypoxia Positron-Emission-Tomography (PET) as well as Computed Tomography (CT) radiomics have been shown to be prognostic for radiotherapy outcome. Here, we investigate the stratification potential of CT-radiomics in head and neck cancer (HNC) patients and test if CT-radiomics is a surrogate predictor for hypoxia as identified by PET. MATERIALS AND METHODS: Two independent cohorts of HNC patients were used for model development and validation, HN1 (n = 149) and HN2 (n = 47). The training set HN1 consisted of native planning CT data whereas for the validation cohort HN2 also hypoxia PET/CT data was acquired using [18F]-Fluoromisonidazole (FMISO). Machine learning algorithms including feature engineering and classifier selection were trained for two-year loco-regional control (LRC) to create optimal CT-radiomics signatures.Secondly, a pre-defined [18F]FMISO-PET tumour-to-muscle-ratio (TMRpeak ≥ 1.6) was used for LRC prediction. Comparison between risk groups identified by CT-radiomics or [18F]FMISO-PET was performed using area-under-the-curve (AUC) and Kaplan-Meier analysis including log-rank test. RESULTS: The best performing CT-radiomics signature included two features with nearest-neighbour classification (AUC = 0.76 ± 0.09), whereas AUC was 0.59 for external validation. In contrast, [18F]FMISO TMRpeak reached an AUC of 0.66 in HN2. Kaplan-Meier analysis of the independent validation cohort HN2 did not confirm the prognostic value of CT-radiomics (p = 0.18), whereas for [18F]FMISO-PET significant differences were observed (p = 0.02). CONCLUSIONS: No direct correlation of patient stratification using [18F]FMISO-PET or CT-radiomics was found in this study. Risk groups identified by CT-radiomics or hypoxia PET showed only poor overlap. Direct assessment of tumour hypoxia using PET seems to be more powerful to stratify HNC patients.

2D and 3D convolutional neural networks for outcome modelling of locally advanced head and neck squamous cell carcinoma.

For treatment individualisation of patients with locally advanced head and neck squamous cell carcinoma (HNSCC) treated with primary radiochemotherapy, we explored the capabilities of different deep learning approaches for predicting loco-regional tumour control (LRC) from treatment-planning computed tomography images. Based on multicentre cohorts for exploration (206 patients) and independent validation (85 patients), multiple deep learning strategies including training of 3D- and 2D-convolutional neural networks (CNN) from scratch, transfer learning and extraction of deep autoencoder features were assessed and compared to a clinical model. Analyses were based on Cox proportional hazards regression and model performances were assessed by the concordance index (C-index) and the model's ability to stratify patients based on predicted hazards of LRC. Among all models, an ensemble of 3D-CNNs achieved the best performance (C-index 0.31) with a significant association to LRC on the independent validation cohort. It performed better than the clinical model including the tumour volume (C-index 0.39). Significant differences in LRC were observed between patient groups at low or high risk of tumour recurrence as predicted by the model ([Formula: see text]). This 3D-CNN ensemble will be further evaluated in a currently ongoing prospective validation study once follow-up is complete.

The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping.

Background Radiomic features may quantify characteristics present in medical imaging. However, the lack of standardized definitions and validated reference values have hampered clinical use. Purpose To standardize a set of 174 radiomic features. Materials and Methods Radiomic features were assessed in three phases. In phase I, 487 features were derived from the basic set of 174 features. Twenty-five research teams with unique radiomics software implementations computed feature values directly from a digital phantom, without any additional image processing. In phase II, 15 teams computed values for 1347 derived features using a CT image of a patient with lung cancer and predefined image processing configurations. In both phases, consensus among the teams on the validity of tentative reference values was measured through the frequency of the modal value and classified as follows: less than three matches, weak; three to five matches, moderate; six to nine matches, strong; 10 or more matches, very strong. In the final phase (phase III), a public data set of multimodality images (CT, fluorine 18 fluorodeoxyglucose PET, and T1-weighted MRI) from 51 patients with soft-tissue sarcoma was used to prospectively assess reproducibility of standardized features. Results Consensus on reference values was initially weak for 232 of 302 features (76.8%) at phase I and 703 of 1075 features (65.4%) at phase II. At the final iteration, weak consensus remained for only two of 487 features (0.4%) at phase I and 19 of 1347 features (1.4%) at phase II. Strong or better consensus was achieved for 463 of 487 features (95.1%) at phase I and 1220 of 1347 features (90.6%) at phase II. Overall, 169 of 174 features were standardized in the first two phases. In the final validation phase (phase III), most of the 169 standardized features could be excellently reproduced (166 with CT; 164 with PET; and 164 with MRI). Conclusion A set of 169 radiomics features was standardized, which enabled verification and calibration of different radiomics software. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Kuhl and Truhn in this issue.

Assessing robustness of radiomic features by image perturbation.

Image features need to be robust against differences in positioning, acquisition and segmentation to ensure reproducibility. Radiomic models that only include robust features can be used to analyse new images, whereas models with non-robust features may fail to predict the outcome of interest accurately. Test-retest imaging is recommended to assess robustness, but may not be available for the phenotype of interest. We therefore investigated 18 combinations of image perturbations to determine feature robustness, based on noise addition (N), translation (T), rotation (R), volume growth/shrinkage (V) and supervoxel-based contour randomisation (C). Test-retest and perturbation robustness were compared for combined total of 4032 morphological, statistical and texture features that were computed from the gross tumour volume in two cohorts with computed tomography imaging: I) 31 non-small-cell lung cancer (NSCLC) patients; II): 19 head-and-neck squamous cell carcinoma (HNSCC) patients. Robustness was determined using the 95% confidence interval (CI) of the intraclass correlation coefficient (1, 1). Features with CI ≥ 0:90 were considered robust. The NTCV, TCV, RNCV and RCV perturbation chain produced similar results and identified the fewest false positive robust features (NSCLC: 0.2-0.9%; HNSCC: 1.7-1.9%). Thus, these perturbation chains may be used as an alternative to test-retest imaging to assess feature robustness.

CT imaging during treatment improves radiomic models for patients with locally advanced head and neck cancer.

BACKGROUND AND PURPOSE: The development of radiomic risk models to predict clinical outcome is usually based on pre-treatment imaging, such as computed tomography (CT) scans used for radiation treatment planning. Imaging data acquired during the course of treatment may improve their prognostic performance. We compared the performance of radiomic risk models based on the pre-treatment CT and CT scans acquired in the second week of therapy. MATERIAL AND METHODS: Treatment planning and second week CT scans of 78 head and neck squamous cell carcinoma patients treated with primary radiochemotherapy were collected. 1538 image features were extracted from each image. Prognostic models for loco-regional tumour control (LRC) and overall survival (OS) were built using 6 feature selection methods and 6 machine learning algorithms. Prognostic performance was assessed using the concordance index (C-Index). Furthermore, patients were stratified into risk groups and differences in LRC and OS were evaluated by log-rank tests. RESULTS: The performance of radiomic risk model in predicting LRC was improved using the second week CT scans (C-Index: 0.79), in comparison to the pre-treatment CT scans (C-Index: 0.65). This was confirmed by Kaplan-Meier analyses, in which risk stratification based on the second week CT could be improved for LRC (p = 0.002) compared to pre-treatment CT (p = 0.063). CONCLUSION: Incorporation of imaging during treatment may be a promising way to improve radiomic risk models for clinical treatment adaption, i.e., to select patients that may benefit from dose modification.

Development and Validation of a Gene Signature for Patients with Head and Neck Carcinomas Treated by Postoperative Radio(chemo)therapy.

Purpose: The aim of this study was to identify and independently validate a novel gene signature predicting locoregional tumor control (LRC) for treatment individualization of patients with locally advanced HPV-negative head and neck squamous cell carcinomas (HNSCC) who are treated with postoperative radio(chemo)therapy (PORT-C).Experimental Design: Gene expression analyses were performed using NanoString technology on a multicenter training cohort of 130 patients and an independent validation cohort of 121 patients. The analyzed gene set was composed of genes with a previously reported association with radio(chemo)sensitivity or resistance to radio(chemo)therapy. Gene selection and model building were performed comparing several machine-learning algorithms.Results: We identified a 7-gene signature consisting of the three individual genes HILPDA, CD24, TCF3, and one metagene combining the highly correlated genes SERPINE1, INHBA, P4HA2, and ACTN1 The 7-gene signature was used, in combination with clinical parameters, to fit a multivariable Cox model to the training data (concordance index, ci = 0.82), which was successfully validated (ci = 0.71). The signature showed improved performance compared with clinical parameters alone (ci = 0.66) and with a previously published model including hypoxia-associated genes and cancer stem cell markers (ci = 0.65). It was used to stratify patients into groups with low and high risk of recurrence, leading to significant differences in LRC in training and validation (P < 0.001).Conclusions: We have identified and validated the first hypothesis-based gene signature for HPV-negative HNSCC treated by PORT-C including genes related to several radiobiological aspects. A prospective validation is planned in an ongoing prospective clinical trial before potential application in clinical trials for patient stratification. Clin Cancer Res; 24(6); 1364-74. ©2018 AACR.

A comparative study of machine learning methods for time-to-event survival data for radiomics risk modelling.

Radiomics applies machine learning algorithms to quantitative imaging data to characterise the tumour phenotype and predict clinical outcome. For the development of radiomics risk models, a variety of different algorithms is available and it is not clear which one gives optimal results. Therefore, we assessed the performance of 11 machine learning algorithms combined with 12 feature selection methods by the concordance index (C-Index), to predict loco-regional tumour control (LRC) and overall survival for patients with head and neck squamous cell carcinoma. The considered algorithms are able to deal with continuous time-to-event survival data. Feature selection and model building were performed on a multicentre cohort (213 patients) and validated using an independent cohort (80 patients). We found several combinations of machine learning algorithms and feature selection methods which achieve similar results, e.g. , MSR-RF: C-Index = 0.71 and BT-COX: C-Index = 0.70 in combination with Spearman feature selection. Using the best performing models, patients were stratified into groups of low and high risk of recurrence. Significant differences in LRC were obtained between both groups on the validation cohort. Based on the presented analysis, we identified a subset of algorithms which should be considered in future radiomics studies to develop stable and clinically relevant predictive models for time-to-event endpoints.

FDG uptake in normal tissues assessed by PET during treatment has prognostic value for treatment results in head and neck squamous cell carcinomas undergoing radiochemotherapy.

BACKGROUND AND PURPOSE: Pronounced early side effects have been suggested to be a positive prognostic factor in patients undergoing chemo-radio-therapy (CRT) for head and neck squamous cell carcinomas (HNSCC). We assessed the utility of positron emission tomography (PET) during treatment to analyze the correlation of 18F-fluorodeoxyglucose (FDG) uptake in off target structures within the irradiated volume with outcome. MATERIAL AND METHODS: Two independent cohorts of patients with locally advanced HNSCC, both treated within prospective clinical imaging trials with curatively intended CRT were retrospectively analyzed. The exploratory cohort included 50, the independent validation cohort 26 patients. Uptake of FDG in mucosa and submucosal soft tissues (MST) as well as in other structures was assessed at week 4 during treatment. Considered endpoints were local tumor control (LC) and overall survival (OS). The prognostic value of FDG uptake on the endpoints was measured by the concordance index (ci) using univariate and multivariate Cox regression analyses based on the continuous variables of the exploratory cohort. RESULTS: In the exploratory cohort FDG uptake in MST was prognostic for LC (hazard ratio HR=0.23, p=0.025) and OS (HR=0.30, p=0.003) in univariate analyses. These findings remained significant upon multivariate testing (LC HR=0.14, p=0.011; OS HR=0.20, p=0.001) and were confirmed in the validation cohort for LC (HR=0.15, p=0.034) and OS (HR=0.17, p=0.003). Also the SUVmean threshold of MST that was generated within the exploratory cohort (2.375) yielded significant differences in OS (p=0.006) and a statistical trend for LC (p=0.078) when applied to the validation cohort. CONCLUSIONS: FDG uptake in normal tissues within the irradiated volume measured by PET during treatment has significant prognostic value in HNSCC. This effect may potentially be of use for personalized treatment adaptation.